Continuous Verification of Large Embedded Software 
using SMT-Based Bounded Model Checking 
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The complexity of software in embedded systems has in- 
creased significantly over the last years so that software ver- 
ification now plays an important role in ensuring the overall 
product quality. In this context, SAT-based bounded model 
checking has been successfully applied to discover subtle 
errors, but for larger applications, it often suffers from the 
state space explosion problem. This paper describes a new 
approach called continuous verification to detect design er- 
rors as quickly as possible by looking at the Software Con- 
figuration Management (SCM) system and by combining 
dynamic and static verification to reduce the state space 
to be explored. We also give a set of encodings that pro- 
vide accurate support for program verification and use dif- 
ferent background theories in order to improve scalability 
and precision in a completely automatic way. A case study 
from the telecommunications domain shows that the pro- 
posed approach improves the error-detection capability and 
reduces the overall verification time by up to 50%. 



1 Introduction 

Embedded computer systems are used in a wide range of 
sophisticated applications, such as mobile phones or set-top 
boxes providing internet connectivity. The functionality de- 
manded for such systems has increased significantly and an 
increasing number of functions are implemented in software 
rather than hardware. As a consequence, the verification of 
the software design and the correctness of its implementa- 
tion have become increasingly difficult. 

Bounded model checking (BMC) has been successfully 
applied to verify embedded software and discovered subtle 
errors in real designs [3|. BMC generates verification con- 
ditions (VCs) that reflect the exact path in which a statement 
is executed, the context in which a given function is called, 
and the bit-accurate representation of the expressions. Prov- 
ing the validity of these VCs remains the main performance 



bottleneck in verifying large embedded software, despite at- 
tempts to cope with increasing system complexity by apply- 
ing SMT (Satisfiability Modulo Theories) solvers |[Tll8l[T2t 

We address this bottleneck with a new concept called 
continuous verification. It aims to automatically detect de- 
sign errors and integration problems as quickly as possi- 
ble by exploiting information from the software configu- 
ration management (SCM) system, systematically focusing 
the verification effort on new or modified functions. We use 
equivalence checking to determine whether modified func- 
tions need to be re-verified formally and we use existing test 
cases to reduce the search space for the model checker, thus 
combining dynamic and static verification. We show that 
the continuous verification approach substantially reduces 
the verification time of large embedded software. 

We also exploit the different background theories of 
SMT solvers and combine different theories and solvers, 
based on an analysis of the syntactic structure of a given 
ANSI-C program. We then demonstrate that this combi- 
nation significantly improves the performance of software 
model checking for a wide range of embedded software 
benchmarks from the telecommunications domain. This 
is the first work that exploits the syntactic structure of an 
ANSTC program in order to combine different encodings 
and SMT solvers for BMC of embedded software. 

We describe how these new (compared to our previous 
work [8 1) contributions are realized in ESBMC, the Effi- 
cient SMT-Based Bounded Model Checker. ESBMC ex- 
tends CBMC I0 to support different theories and different 
SMT solvers and to make use of high-level information to 
simplify and reduce the unrolled formula size. Experimen- 
tal results show that our approach scales significantly bet- 
ter than both the SAT-based and SMT-based versions of the 
CBMC model checker. In addition, the continuous verifi- 
cation approach and the combination of different encodings 
and solvers allow us to go deeper into the system (compared 
to software model checkers only) and explore more exhaus- 
tively the state space (compared to testing only). This hy- 
brid solution is suitable for checking properties in large state 
spaces. 



2 Continuous Verification 

Our approach has its roots in the continuous integration 
(CI) practice described by Fowler [111 . CI relies on every 
developer to create and execute unit, functional and inte- 
gration tests before committing their source code to a sin- 
gle source repository. It also assumes the existence of an 
automated unit test framework. The SCM is then used to 
perform the system build and test processes in a completely 
automatic way. In continuous verification, we use the same 
information (i.e., development history and test cases) in a 
different way to improve the coverage and substantially re- 
duce the verification time throughout the development of a 
product or product line. We use SMT-based bounded model 
checking to verify for each system build that the entire sys- 
tem still satisfies all properties given as assertions by the de- 
signers, as well as a range of language-specific safety prop- 
erties such as the absence of arithmetic under- and overflow, 
out-of-bounds array indexing, or nil-pointer dereferencing. 
Figure Q] shows the main elements and steps of the con- 
tinuous verification approach; the gray boxes indicate core 
steps. Section [3] describes the software verification process 
in more detail. 




Figure 1. Continuous Verification 

For large embedded software, the computational effort 
to re-verify the entire software from scratch is too high, 
and is largely wasted if, as is often the case, the changes 
are small ifTTl . For each system build, we thus consult 
the SCM to identify the functions and methods that have 
actually been modified and focus on these. We then use 
equivalence checking to determine whether they need to be 
re-verified formally: if we can prove that the old and new 
versions of a function are functionally equivalent, then we 
do not need to show for the new version any of the prop- 
erties already shown for the old version. This reduces the 
immediate verification effort because proving the equiva- 
lence of two function versions is often less expensive than 



unsigned signallnverter ( int signal) { 
unsigned inverter; 
if ( signal >= 0) 

inverter = signal ; 
else 

inverter = — l*signal; 
return inverter ; 

} 



(a) 



unsigned si 


gn al In ver t er ( in t signal) { 


if( signal 


< 0) 


return 


-signal ; 


else 




return 


signal ; 


} 





(b) 



Figure 2. (a) Original function to invert the 
sign of signal, (b) Optimized version. 



re-verifying the function. However, and more importantly, 
it also reduces overall system verification efforts because it 
limits the propagation of changes through the system: if we 
can prove the two versions of the function computationally 
equivalent, then we do not need to re-verify any other func- 
tion that depends it (unless that function has been changed 
as well). Of course, proving the equivalence of two func- 
tions is in general undecidable, due to unbounded memory 
usage [16], and the effort we spend in trying to do so might 
be wasted. However, in our experience, this does not occur 
very often, since a high degree of predictability is a desir- 
able design characteristic in embedded software (i.e., dy- 
namic memory allocations and recursion are discouraged). 

As an example, consider the two versions of the signal- 
Inverter function shown in Figure [2] They were extracted 
from the embedded software of two releases of a medical 
device product. In order to prove the equivalence of these 
two ANSI-C functions, we compare their input-output re- 
lations. We thus first (i) remove from each function the 
variable declarations and return statements, ( ii) convert the 
function bodies into single static assignment (SSA) form 
IfTTl (i.e., we introduce fresh variables by subscripting the 
original name such that every assignment has a unique left 
hand side), and (Hi) conjoin all program statements. These 
operations produce two intermediate formulas oi\ and a 2 
representing the functions' computations, as shown below. 

inverteri — signa^ 

A inverter 2 = — 1 * signal^ 

A inverter^ = (signa^ > ? inverteri : inverter-i) 
signal' 2 — {signal^ < ? —signal^ : signal^) 



For the actual equivalence check, we identify the input 
variables (i.e., signal ± — signal^), and show using SMT- 
based bounded model checking that, given the representa- 
tion of the function bodies, the output variables then also 
coincide: 

(a i A Q!2 A (signal^ = signal'^y.^inverter^ = signal^) 

2.1 Specifying Temporal Properties with 
Biichi Automata 

In addition to the language-specific safety properties, we 
can also show user-specified properties. These can be given 
directly as assertions in the code, using C's assert macro 
to state an assumption, or as formulas in linear-time tempo- 
ral logic (LTL), which can track temporal properties of the 
software design. We translate the LTL formulas into Biichi 
automata using the Wring tool [21] and further into ANSI- 
C and merge them into the code. The resulting ANSI-C 
program then monitors the design's progress and watches 
out for violations of the specified properties (as described 
in Section[3]l. 

As an example, we extract two properties from the spec- 
ification of the medical device, and show how they can be 
modelled and used in the context of the continuous verifica- 
tion. The device, called a pulse oximeter [7|, is responsible 
for measuring the oxygen saturation (SpC»2) and heart rate 
(HR) in the blood system using a non-invasive method. In 
particular, we verify (a) the data flow to compute the HR 
value that is provided by the pulse oximeter sensor hard- 
ware and (b) whether the user of the pulse oximeter is ca- 
pable of adjusting the sample time of the embedded device. 
The properties (a) and (b) can be expressed using the fol- 
lowing LTL pattern: 

AG{p^Fr) (1) 

Here, A ("for all paths"), G ("always"), and F ("eventu- 
ally") are the LTL quantifiers, and p and r represent the 
required pre- and post-states. In the example, for the prop- 
erty (a), p denotes the state that the buffer contains HR and 
SpC>2 raw data, while r denotes the state that defines the re- 
spective HR value. Consequently, any state containing the 
HR and Sp02 raw data in the buffer is eventually followed 
by a state representing the respective HR value. 

A Biichi automaton is a finite automaton over infinite 
words. It differs from a standard finite automaton over fi- 
nite words in the definition of accepting a word, which is 
based on passing through an accepting state infinitely of- 
ten (rather than terminating in a final state) [5 1. The Biichi 
automata we consider here work over computation traces, 
i.e., sequences of states of the program to be analyzed. 
These are abstracted by the predicates of interest (here p and 
r). Hence the "words" can be represented by sequences of 



Buechi Automata ^ ANSI-C Monitor 




()%2; 



Figure 3. Specifying Temporal Properties for 
Software. 



propositional expressions over the variables p and r. Fig- 
ure [3] shows the non-determininistic Biichi automaton that 
represents the LTL formula ([]]) and its corresponding ANSI- 
C monitor. The transition function 5 is given by the follow- 
ing table. 
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From the initial state, we can transition to S3 if rV^p holds, 
stay in the initial state if either r, or -rp holds, or nondeter- 
ministically transition to either S 1 or S2 if none of the three 
properties hold (denoted by 1). This automaton will accept 
all infinite words that represent computations in which each 
state in which p holds will eventually be followed by a state 
in which r holds. In order to model the nondeterministic 
transition of the Biichi-automata in the ANSI-C specifica- 
tion, we use a function nondetjiint{) (which returns any 
number of type unsigned int) and then restricts its returning 
value to the domain {0, 1}. The property is then checked by 
using a "monitor" in such a way that the C program moni- 
tors the design's progress and watches out for a specific type 
of error up to the bound k. An assertion is then used to claim 
that an error is never encountered. In our example, we en- 
sure that the buffer is not empty and contains the computed 
HR and SpC»2 values. 

2.2 Generalizing Test Cases 

After detecting new and/or modified functions, we use 
the existing unit test cases to reduce the state space to be ex- 
plored by the model checker. In this phase, we first run the 
unit tests, keeping track of which inputs have already been 



used. We then guide the model checker to visit states that 
have not been visited previously. In addition, the test cases 
also help to reduce the state space to be explored in another 
way: by using the test stubs, we can break the global model 
(containing the entire program) into local models (contain- 
ing only the functions under test) and generate on-demand 
the reachable states to be visited by the model checker, start- 
ing with the state described by the test case. We can so 
reduce the number of paths and variables to be considered 
during model checking. 

As an example consider the three simple C functions 
shown in Figure [4] that were also extracted from the pulse 
oximeter embedded software, and one of the test cases 
shown in Figure [5] The functions implement a simple cir- 
cular buffer using a FIFO (First In, First Out) policy. The 
test case checks whether messages are correctly added to 
and removed from the circular buffer using the FIFO policy. 
Other test cases check for buffer underflow and/or overflow 
and whether the elements are lost before reading them from 
the buffer. 



static char buffer [BUEFERJVIAX] : 
void initLog(int max) { 

buffer_size = max ; 

first = next = 0; 

} 

int removeLogElem ( void ) { 
first + + ; 

return buffer[first — 1]; 

} 

void insertLogElem ( int b) { 
if (next < buffer.size ) { 
buffer [ next ] = b ; 
next = ( next + l)% b uf f e r _siz e : 
assert(next<buffer_size ); 



} 



} 



static void te s t Circul arB uf f er ( void ){ 

int senData [] = {1 , -128, 98, 88, 59, 
1, -128, 90, 0, -37}; 

int i ; 

initLog ( 5 ) ; 
for(i=0; i < 10; i++) 

insertLogElem(senData[i ]); 
for(i=5; i <10; 

TEST_ASSERT_EQUAL_INT ( senData [ i ] , 

removeLogElem ( ) ) : 



Figure 5. A unit test for the functions shown 
in FigurelH 



line 12 in Figure 0J. However, in general this approach 
can lead to false positives because the non-deterministic 
choice of values for program variables may force the ex- 
ploration of paths that are infeasible in the original pro- 
gram. Rather than modifying the program we thus modify 
the test stubs and replace the concrete input values by non- 
deterministic choices. Here, we replace the initialization 
of the array senData (see line 2 and 3 of Figure [5]) by int 
senData[] = {nondetJntQ,. . . ,nondetJnt()}. We then use 
assMme-statements to force the model checker away from 
the values that have already been explored during testing. 
In order to block larger parts of the search space, we use 
the given concrete values from all stubs and combine the 
respective values into a single interval for each variable or 
array element; here we assume that all "obvious" bound- 
ary values are used in some of the stubs, so that we are not 
blocking parts of the search space contain "obvious" errors 
but force the model checker towards the "unobvious" er- 
rors. In the example, we thus add an assMme-statement such 
as assume(senData[0]<=l && senData [0]>=42) and are 
then able to find two bugs related to overflow and under- 
flow. 



Figure 4. Implementation of a circular buffer. 

The pulse oximeter sources contain seven test cases, 
which intend to cover all possible execution paths related 
to the circular buffer, and during dynamic verification, we 
are not able to find any bug in the circular buffer implemen- 
tation with these. However, the implementation is flawed: 
the array buffer is declared to be of type char[] (see line 1 in 
Figure|4]i but we assign an element b of type int (see line 14) 
The test cases do not uncover this error because they happen 
to use only integer values that can safely be cast to a char. 

Using SMT-based BMC, we can detect this bug by 
non-determininistically assigning a value to the parame- 
ter b (i.e., by adding an assignment b = nondetJnt() after 



3 SMT-based Bounded Model Checking of 
Embedded ANSI-C Software 

In BMC, the program to be analyzed is modelled as a 
state transition system, which is built by extracting its be- 
haviour from the control-flow graph (CFG) ifTTl . This graph 
is used as part of a translation process from program text to 
SSA-form. A node in the CFG represents either a deter- 
ministic or non-deterministic assignment or a conditional 
statement, while an edge in the CFG represents the ability 
of the program to change the control location. 

A state transition system M = (S, Sq,j) is an abstract 
machine that consists of a set of states S (where So C S 



represents the set of initial states) and 7 C S x S is the 
concrete transition relation Q. A state s E S consists of 
the value of the program counter pc and the values of all 
program variables. An initial state sq assigns the initial pro- 
gram location of the CFG to the pc. We identify each tran- 
sition 7 = (s;,Sj + i) between two states Si and s,; + i with 
a logical formula j(si, s;+i) that captures the constraints 
on the corresponding values of the program counter and the 
program variables. 

Given a transition system M, a property <fi, and a bound 
k, BMC unrolls the system k times and translates it into 
a verification condition tp such that ip is satisfiable if and 
only if <f> has a counterexample of depth less than or equal 
to k. The VC ip is a quantifier-free formula in a decidable 
subset of first-order logic, which is then checked for satis- 
fiability by an SMT solver. In this work, we are interested 
in checking two classes of properties: safety and liveness. 
The model checking problem associated with SMT-based 
BMC for checking safety properties is then formulated by 
constructing the following logical formula |[Tl [T2l : 



k-l 



property 

i) k =/(s ) A f\ 7 (*,«(+!) A -vfa (2) 



constraints 

where <f>Q represents a safety property (pc in step k, I is the 
function for the set of initial states of M and 7 (sj, s;+i) is 
the function of the transition relation of M at time steps i 
and i + 1. Hence, the formula 7 (sj, s i+ i) unrolls the 
transition system and then represents the set of all execu- 
tions of M up to the length k or less. ~^(f> G represents the 
condition that (pQ is violated by a bounded execution of M 
of length k or less. A counterexample for a property <pc is 
a sequence of states so, s\, . . . , Sk with so £ So, Sfc E S, 
and 7 (sj, Sj+i) for < i < fe. Liveness properties cpp are 
checked by encoding ~^<p F in a loop within a bounded exe- 
cution of length at most k, such that <pp is negated on each 
state in the loop [3 |. In this case, formula (|2]i is rewritten as: 



fc-i 



i> k =I( SQ )A /\ 7 ( Si)5i+1 )A 



(3) 



i=0 



For further information about the main software compo- 
nents of our ESBMC tool, we refer the reader to JSJ. 

3.1 Optimizations 

ESBMC implements some standard code optimization 
techniques such as constant folding, forward substitution, 
and reduction of variables ifTTl . In our previous work |8|, 
we described how constant folding and forward substitution 
are implemented in the context of bounded model checking. 



int main ( ) { 




int a [2] , i, x; 




if (x==0) 




a[i]=0; 




else 




a[ i +2] = 1; 




assert ( a [ i + 1 ] = = 


= D; 


} 




(a) 


g l == (xl == 0) 




al == (aO WITH 


[i0:=0]) 


a2 == aO 




a3 == (a2 WITH 


[2+iO : = 1]) 


a4 == (gl ? al 


: a3) 


tl == (a4[l+i0] 


== 1) 



(b) 



Figure 6. (a) A C program with violated prop- 
erty, (b) The C program of (a) in SSA form. 



Here, we focus on how to reduce the number of redundant 
variables generated from the BMC instances. 

We use the code in Figure [6] as a running example. Fig- 
ure |6ja) shows a syntactically valid C program that acci- 
dentally writes to an address outside the allocated memory 
region of the array a (line 6). Figure |6jb) shows the pro- 
gram in SSA form, where it only consists of ;/ instructions, 
assignments and assertions. As we can see in Figure HJb), 
for each assignment of the form a = expr (lines 4 and 6 from 
Figure|6ja)), the left-hand side variable is replaced by a new 
variable (e.g., ax, 0,3). In addition, the condition in line 3 
is replaced by a guard (e.g., gi), the array index i in lines 3, 
6 and 7 is replaced by iq (note that the variable i is not mod- 
ified in the C program of Figure |6j a)) and the assert macro 
in line 7 is replaced by a Boolean variable t\. From this 
representation, we build the constraints (C) and properties 
(P) formulae shown in (0]l and (0 using the quantifier-free 
fragment of the first-order theories (e.g., linear arithmetic, 
arrays), which are then passed to an SMT solver as CA^P 
to check satisfiability. 



C := 



P := 



9i ■= (xx = 0) 

A a± := store(ao, «0i 0) 

A a 2 := a 

A 03 := store(d2, 2 + io, 1) 
A a 4 := ite(gx,ax,a 3 ) 

io > A i a < 2 
A2 + i >0A2 + i o <2 
A1 + j >0A1 + ! O <2 
A select(a,4, io ■ ' ' 



(4) 



1) 



1 



(5) 



However, the SSA transformation in the CBMC front- 
end introduces redundant variables, which creep into the 
formula passed into the SMT-solver, thus increasing the 
search space. Figure |6]^b) shows an example of an addi- 
tional (redundant) variable (i.e., 02) created in order to keep 
the value of array a before the statement in line 3 of Fig- 
ure [6{ a). In our back-end, we eliminate functionally deter- 
mined variables by substitution. In our running example, 
a-2 == ao holds in all states, so we can eliminate this con- 
straint and the variable 02 by substituting every occurrence 
of a,2 by ao. we thus rewrite (01 as: 

" g x := ( Xl = 0) 
c . = A ci! := store(a ,i 0l Q) 

A 0,3 := store(ao, 2 + iq, 1) 
A 0,4 :— ite(g\, 0,1,0,3) 

Eliminating functionally determined variables allows us to 
remove most of the intermediate variables generated dur- 
ing the symbolic execution and consequently the size of the 
final formula to be sent to the SMT solver. 

3.2 Encodings 

In this section, we focus on new encodings (compared to 
our previous work ||8]) that allow us to analyze more com- 
plex ANSI-C programs and show how to exploit different 
data types and solvers to speed up the verification process. 

3.2.1 Fixed-Point Arithmetic 

Embedded applications from discrete control and telecom- 
munications domains often require arithmetic on non- 
integral numbers. However, full floating-point arithmetic 
is too heavyweight to be encoded into the BMC framework; 
instead, we approximate it by fixed-point arithmetic and use 
two different representations: in base 2 (when dealing with 
bit-vector arithmetic) and in base 10 (when dealing with ra- 
tional arithmetic). Hence, in fixed-point arithmetic, we en- 
code the numbers using the integer part and fractional part 
(i.e., after the radix point ".") [16|. For instance, the num- 
bers 0.75 and 0.125 are fixed-point numbers with two-digit 
and three-digit fractional parts respectively. The number 
0.75 can be represented as (0000.11) in base 2 or 3/4 in base 
10 while the number 0.125 can represented as (0000.0010) 
in base 2 or 125/1000 in base 10. Given a rational number 
R that consists of an integer part / with m bits and a frac- 
tion part F with n bits, we represent R by I.F and denote 
it by (I.F)/2 n . 

Using bit-vector arithmetic, we encode fixed-point arith- 
metic exactly as in binary encoding and assume that the 
numbers have the same bitwidth before and after the radix 
point. If the numbers do not have the same bitwidth (e.g., if 
we want to sum 0.75 and 0. 125), we add zeros from the right 



in the fractional part if there are bits missing after the radix 
point (e.g., 0.75 + 0.125 = (0000.1100) + (0000.0010)) or 
we add bits from the left using sign-extension if there are 
bits missing before the radix point. On the other hand, us- 
ing rational arithmetic, we encode fixed-point numbers by 
mapping {0, i} m+ ™ _ > Q, which extracts the integer part 
and divides the fraction part (F) /2™ and finally converts 
the resulting number to a rational one in base 10. As a re- 
sult, the arithmetic operations are performed in the domain 
of Q instead of M and there is no need to add missing bits to 
the integer and fractional parts. In general, the drawback is 
that some numbers are not precisely represented with fixed- 
point arithmetic. However, we have not detected any false 
positives due to use of fixed-point arithmetic in any of our 
benchmarks. 

3.2.2 Dynamic Memory Allocation 

Although dynamic memory allocation is discouraged in em- 
bedded software, ESBMC is capable of model checking 
programs that use it through the ANSI-C functions mal- 
loc and free. ESBMC checks two properties related to dy- 
namic memory allocation: (i) whether the argument to any 
malloc, free, or dereferencing operation is a dynamic ob- 
ject (ISJ)YNAMICjOBJECT) and (ii) whether the argument 
to any free or dereferencing operation is still a valid object 
{VALID .OBJECT). 

Formally, let p a be a pointer expression that points to the 
object o of type t and let a be an array of type t and size 
n, where n is of integer type and represents the number of 
elements to be allocated. In our encoding, each dynamic 
object has an unique identifier denoted by pk, where the 
subscript k indicates the objects "serial number" in sequen- 
tial order of all dynamically created objects. Let i be an 
integer variable that indicates the position in which the ob- 
ject pointed to by p must be stored in array a. We thus 
encode IS .DYNAMIC .OBJECT as a literal l d with the fol- 
lowing constraint: 

h ( V/ P" = P°) A (° ^ * < n ) ( ? ) 

\n=0 / 

To check for invalid objects, we add one additional bit 
field v to each dynamic object to indicate whether it is still 
alive or not. Let _L denote the state that the object is not 
alive. We then encode WALID .OBJECT as a literal l v with 
the following constraint: 

lv {Vo-y ± J-) (8) 

3.2.3 Exploiting Datatypes and Solvers 

Modern SMT solvers provide ways to model the program 
variables as bit-vectors or as elements of a numerical do- 



main (e.g., Z, Q, or R). If the program variables are mod- 
elled as bit-vectors of fixed size, then the result of the anal- 
ysis can be precise (w.r.t. the ANSI-C semantics), depend- 
ing on the size considered for the bit- vectors. On the other 
hand, if the program variables are modelled as numerical 
values, then the result of the analysis is independent from 
the actual binary representation, but the analysis may not be 
precise when arithmetic expressions are involved. As a mo- 
tivating example, consider the following small C program 
from 1 6 1 as shown in Figure Q 



int main() { 

unsigned char a, b; 
unsigned int result=0, i; 
a=nondet_uchar (); 
b=nondet_uchar (); 
for(i=0; i <8; i ++) 
if ((b»i)&l) 

result +=(a«i ) ; 
assert ( result==a*b); 

} 



Figure 7. A C program that uses shift-and-add 
to multiply two numbers. 

This program nondeterministically selects two values of 
type unsigned char and uses bitwise AND, right- and left- 
shift operations to multiply them. Reasoning about this pro- 
gram by means of integer arithmetic produces wrong results 
if the bit-level operators are treated as uninterpreted func- 
tions (UFs). Although UFs simplify the proofs, they ignore 
the semantics of the operators and consequently make the 
formula weaker. In addition, the majority of the software 
model checkers (e.g., SMT-CBMC fl] and BLAST |[T4l ) 
fail to check the assertion in line 9. On the other hand, bit- 
vector arithmetic allows us to encode bit-level operators in a 
more accurate way. However, in our benchmarks, we noted 
that the majority of VCs are solved faster if we model the 
basic datatypes as integer and/or real. Consequently, we 
have to trade off between speed and accuracy which might 
be two competing goals in formal verification using SMT. 

Based on the extent to which the SMT solvers support 
the domain theories and on experimental results obtained 
with a large set of benchmarks, we developed a simple but 
effective heuristic to determine the best representation for 
the program variables as well as the best SMT-solver to be 
used in order to check the properties of a given ANSI-C 
program. Our default representation for encoding the con- 
straints and properties of ANSI-C programs are integers and 
reals, respectively, and our default solver is Z3. We then ex- 
plore the CFG representation of the program. If we find 
expressions that involve bit operations (e.g., <<, >>, &, |, 
©) or typecasts from signed to unsigned datatypes and vice- 



versa, we encode the corresponding variables as bit-vectors 
and switch the SMT solver either to Boolector (if no point- 
ers are used) or Z3 (if pointers are used). 

4 Experimental Evaluation 

The experimental evaluation of our work consists of 
three parts. Section 14.11 contains the results of applying 
ESBMC to the verification of six sets of ANSI-C bench- 
marks. The purpose of this first part is to check the error- 
detection capability of ESBMC since most of these bench- 
marks contain ANSI-C programs with and without bugs. 
Section 14.21 contains the experimental results of applying 
ESBMC and CBMC to the verification of embedded soft- 
ware used in the telecommunications domain. The purpose 
of this second part is to evaluate ESBMC's relative perfor- 
mance against CBMC using embedded software industrial 
applications. Section l-Ol contains the results of applying the 
continuous verification approach, described in Section|2] to 
large embedded software used in a commercial product. 

All experiments were conducted on an otherwise idle In- 
tel Pentium Dual CPU, 2GHz with 4 GB of RAM running 
Linux OS. For all benchmarks, the time limit has been set 
to 3600 seconds for each individual property. All times 
given are wall clock time in seconds as measured by the 
unix time command through a single execution. 

4.1 Error-Detection Capability 

As a first step, we analyze to which extent ESBMC is 
able to handle standard ANSI-C benchmarks. For this pur- 
pose, we analyzed the programs in the VERISEC, NECLA, 
SNU-RT, PowerStone, and WCET benchmarks. VERISEC 
and NECLA are not related to embedded software, but 
they allow us to check the error-detection capability of the 
model checkers since these benchmarks provide ANSI-C 
programs with and without bugs including dynamic me- 
mory allocation, interprocedural dataflow, aliasing, and 
string manipulation. The remaining three benchmarks con- 
tain ANSI-C constructs found in embedded software. 

Table Q] summarizes the results of applying ESBMC to 
the verification of the standard ANSI-C benchmarks; a de- 
tailed description of these experimental results is avail- 
able at users. ecs. sot on. ac.uk/lcc08r/ esbmc. 
Here, #L denotes the total number of lines of code, B de- 
notes the unwinding bound, and #P denotes the number of 
properties to be verified for each ANSI-C program. The 
Time columns contain the total time in seconds to check all 
properties of a given ANSI-C program, broken down into 
frontend (i.e., encoding) and backend (i.e., decision proce- 
dure). Failed indicates how many properties failed during 
the verification process; properties can fail for two reasons: 
either due to time out (TO) or due to memory out (MO). 
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VERISEC 


9090 


2148 


190.85 


228.35 


419.2 


1946 


202 





2 


NECLA 


1011 


208 


59.28 


88.9 


148.18 


188 


20 





3 


SNU-RT 


3102 


790 


3166.94 


12.18 


3179.12 


762 


28 





4 


WCET 


3430 


726 


72.72 


8.28 


81 


722 


4 





5 


POWERSTONE 


2957 


2053 


127.36 


913.64 


1041 


2043 


10 






Table 1. Results of the error-detection capability with ESBMC. 



As mentioned previously, benchmarks VERISEC and 
NECLA contain ANSI-C programs with and without bugs 
and all inputs are replaced by nondeterministic one. In 
these benchmarks, ESBMC is able to detect common de- 
sign errors related to buffer overflow, aliasing, dynamic me- 
mory allocation, and string manipulation. In the remaining 
benchmarks, ESBMC is able to model check ANSI-C pro- 
grams that involve tight interplay between non-linear arith- 
metic, bit operations, pointers and array manipulations. In 
addition, ESBMC was able to find undiscovered bugs in the 
SNU-RT, WCET, and PowerStone benchmarks related to 
arithmetic overflow, invalid pointer and pointer arithmetic. 
We also checked the effect of eliminating functionally de- 
termined variables and we observed that the total verifica- 
tion time can only be reduced by 0.3% to 1% in the bench- 
marks. We believe that the SMT solvers already eliminate 
the redundant variables during the pre-processing phase. 

4.2 Comparison to CBMC 

In order to evaluate ESBMC's performance relative to 
CBMC, we analyzed the embedded software used in a com- 
mercial product from NXP semiconductors [18], a set-top 
box that is used in high definition internet protocol (IP) and 
hybrid digital TV applications. The embedded software of 
this platform relies on the Linux operating system and use 
the LinuxDVB, DirectFB and ALSA applications. 

We evaluated CBMC version 3.3.2 and we invoked both 
tools (i.e., CBMC and ESBMC) by setting manually the file 
name, the unwinding bound and the overflow check. CBMC 
has support for SAT and SMT solvers in the back-end and 
in our comparison we use the SMT solver Z3 for evaluating 
both tools CBMC and ESBMC. Table FJ shows the results 
when applying SAT-based CBMC, SMT-based CBMC and 
ESBMC to the verification of the embedded applications. 
SAT-based CBMC performs better than SMT-based CBMC 
since it is able to analyze one additional embedded applica- 
tion (viz. exStbCc). 

CBMC is not able to check the modules exStbKey, 
exStbFb, and exStbDemo due to memory limitations as well 



as modules exStbResolution and exStbHDMI due to time 
out and segmentation fault. Both CBMC and ESBMC fail 
to model check the module exStbDemo, which contains 
approximately 31KLOC and represents the largest mod- 
ule that we analyzed. However, the results indicate that 
ESBMC scales significantly better than CBMC for prob- 
lems that involve tight interplay between non-linear arith- 
metic, bit operations, pointers and arrays. 

4.3 Scalability 

In order to model check the exStbDemo module we apply 
the continuous verification approach as described in Sec- 
tion[2l Table |3] summarizes the results. Here, #TC denotes 
the total number of test cases. For the dynamic verifica- 
tion, we use the EmbUniQ unit test framework, which pro- 
vides means to apply assertion-based verification in embed- 
ded software written in C through a set of macros to assert 
strings, integers, pointer value, and conditions. 

As described in Section 14.21 the state-of-the-art model 
checkers fail to verify the properties of the exStbDemo ap- 
plication due to memory limitations. However, if we use 
the test cases to guide the state space exploration, we can 
not only define the functions, but also the state variables 
that were not yet fully explored during dynamic verifica- 
tion. As a result, in the function commandLoop, ESBMC 
finds a property violation related to an invalid pointer in 
4.39 seconds using an unwinding bound of 6. However, 
we are not able to increase further the unwinding bound of 
functions commandLoop and getCommand due to memory 
limitations. 

We had access, from the NXP development team, to four 
different product releases (PRs) that contain the applica- 
tion exStbDemo. Based on these four PRs, we identified 
the functions and methods that have actually been modi- 
fied and focus on these since the computational effort to 
re-verify each system build from scratch is too high. The 
four PRs are shown on the right-hand side in Table [3] as PR 
10, 11, 12, 13. The development time of each PR is about 

^mbUnit. http : / /embunit . sourcef orge . net / 
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exStbLED 


430 
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1781 


1817.61 


59 
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exStbHwAcc 


1432 


1000 


115 


0.146 


2.44 


115 








0.71 


3.05 
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0.013 
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115 
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exStbResolution 


353 
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32 
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TO 
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1179.34 


1596.62 


32 
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exStbFb 
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exStbCc 
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31.24 


46.13 
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17 
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MO 


MO 








267 


MO 
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267 



Table 2. Results of the comparison between CBMC and ESBMC. Time-outs are represented with TO 
in the Time column; Examples that exceed available memory are represented with MO in the Time 
column; Internal errors in the tool are represented with f in the Time column. 
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getCommand 
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20 
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314.38 


169.75 


169.45 


298.11 



Table 3. Results of the continuous verification approach. 



one month and each one contains new features, enhance- 
ments (through refactoring), and bug fixes. We use PR10 
as a reference (and starting point) to compare with PR11, 
PR11 to compare against PR12 and so on. The functions 
2-8 shown in Table [3] do not present input/output relations 
and the function getCommand is not equivalent in PR10, 
PR11 and PR12. However, if we compare PR10 to PR11 
and PR12, we can reduce the verification time by up to 50% 
since functions checkEndOflPStream and checkEndOfMe- 
diaStream are not modified in four different versions. In 
addition, the function commandLoop, which represents one 
of the hardest functions, is only modified in PR13. 

5 Related Work 

One way of tackling large verification problems is to 
leverage both parallelism and search diversity fl5l . Holz- 
mann et al. describe the Swarm tool that allows to use differ- 



ent search strategies on multi-core machines fl5l . It is the 
main interface to the SPIN model checker to verify larger 
systems. This approach, however, involves large communi- 
cation overhead and does not take into account information 
from the SCM system in order to focus the verification ef- 
fort on new and/or modified functions. 

Peled proposes a set of combinations between model 
checking and testing, which includes black box checking, 
adaptive model checking, and unit checking [ 19 1. However, 
he does not consider the development history from the SCM 
system and also uses explicit model checking based on au- 
tomata theory, which does not scale well due to the number 
of program variables and data type widths iflOl . In addition, 
Peled only describes the techniques, but does not apply it to 
any commercial product. Gunter and Peled fl3l extend this 
approach by proposing a symbolic verification approach for 
a unit of code, also called unit checking. The authors, how- 
ever, apply this approach only to check whether a complex 



number diverges to infinity, while we focus on the verifi- 
cation of large embedded software. Sen proposes an ap- 
proach to execute a program concretely and symbolically 
by combining random testing and symbolic execution |20ll . 
This approach, however, might fail to compute concrete val- 
ues that satisfy a given path constraint due to the constraint 
solver performance. 

SMT-based BMC is gaining popularity in the formal ver- 
ification community due to the advent of sophisticated SMT 
solvers built over efficient SAT solvers |HJ|9l- Ganai and 
Gupta describe a verification framework for BMC which 
extracts high-level design information from an extended fi- 
nite state machine (EFSM) and applies several techniques 
to simplify the BMC problem lfl2l . However, the authors 
flatten the structures and arrays into scalar variables in such 
a way that they use only the theory of integer and real arith- 
metic, which does not reflect precisely the ANSTC seman- 
tics. Armando et al. also propose a BMC approach using 
SMT solvers for ANSTC programs 0J. I n this approach, 
however, they only make use of linear arithmetic (addition 
and multiplication by constants), arrays, records and bit- 
vectors and as a consequence, their SMT-CBMC prototype 
does not address important constructs of the ANSTC lan- 
guage (e.g., non-linear arithmetic and bit-shift operations). 

Recently, a number of static checkers have been devel- 
oped in order to trade off scalability and precision. Ca- 
lysto is a static checker that is able to verify VCs related 
to arithmetic overflow, nil-pointer dereferencing and asser- 
tions specified by the user J2j . The VCs are passed to the 
SMT solver SPEAR which supports bit-vector arithmetic 
and is customized for the VCs generated by Calysto. How- 
ever, Calysto does not support float-point operations and un- 
soundly approximates loops by unrolling them only once. 
As a consequence, soundness is relinquished for perfor- 
mance. Saturn is another efficient static checker that scales 
to larger systems, but with the drawback of losing precision 
by supporting only the most common integer operators and 
performing at most two unwindings of each loop [22|. 

6 Conclusions 

In this work, we have defined a new concept called con- 
tinuous verification and we have applied it to the verifica- 
tion of large embedded software used in the telecommuni- 
cations domain. In addition, we have described a new set 
of encodings that allow us to reason accurately about em- 
bedded software by discovering subtle bugs in several in- 
dustrial applications and to scale for large embedded soft- 
ware. Furthermore, the experimental results show that our 
SMT-based model checker (ESBMC) outperforms both the 
SAT-based and SMT-based CBMC model checker if we 
consider the verification of embedded software. As a result, 
our approach represents a promising direction to improve 



the state space coverage and to verify quickly the negation 
of properties in larger state spaces. 
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